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ABSTRACT 

These  Research  Notes  form  part  of  a  series  of  notes  extracted  from  work  undertaken  by 
Innovation  Science  in  the  establishment  of  Openness  and  Evolvability  assessment 
Methods  and  Processes.  This  set  of  Research  Notes  focusses  on  Standards  Assessment. 
This  work  was  undertaken  from  the  late  1990s  to  2007  and  focussed  on  the  application 
to  Submarine  Combat  Systems. 
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1.  Introduction 


These  Research  Notes  have  been  extracted  from  work  undertaken  by  Innovation  Science 
under  contract  to  Defence  Science  and  Technology  Group  during  the  period  from  the  late 
1990s  until  early  2007. 

In  entirety  the  Research  Notes  form  a  subset  of  the  overall  assessment  Methodology  and 
Processes  developed  to  assess  system  level  Openness  and  Evolvability. 

The  Research  Notes  within  this  report  focus  on  Standards  Assessment. 


2.  Standards  Assessment 

For  the  purposes  of  the  openness  assessment,  the  term  "standard"  is  used  broadly  to  refer 
to  any  public,  industry  (de  facto),  vendor,  and  custom  specification  that  form  the  rules  or 
model  under  which  a  component  of  the  system  is  implemented  or  governed.  Both  formal 
and  informal  "standards"  are  equally  relevant  to  the  assessment  and  need  to  be  evaluated. 

Each  standard  or  group  of  standards  must  be  assessed  for  their  risk  to  openness  with 
respect  to  their  use  within  the  target  architecture,  infrastructure  or  interface.  This 
standards  assessment  section  is  used  as  part  of  the  architecture,  infrastructure  and 
interface  assessment  process.  The  standard  assessment  is  not  intended  to  assess  the 
suitability  of  a  standard  for  a  particular  application  other  than  the  standard's  effect  on 
openness.  For  example,  a  standard  may  still  pass  the  openness  assessment,  but  not  be  at  all 
suitable  for  use  in  a  particular  real-time  deployment.  The  suitability  of  a  standard  to  a 
particular  engineering  solution  is  beyond  the  scope  of  the  openness  assessment. 

Note  that  if  a  standard  (or  group  of  standards)  is  encapsulated  entirely  within  a  granule 
(i.e.  not  part  of  an  infrastructure  or  interface  definition  used  by  the  granule)  and  is  not 
visible  to  other  granules  within  the  architecture,  then  the  standard  does  not  require 
assessment. 

The  flowchart  shown  in  Figure  1  summarises  the  assessment  process.  Each  question  within 
the  flowchart  is  explained  in  greater  detail  in  the  sections  that  follow. 
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Figure  1.  Assessing  the  Openness  of  a  Standard 
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2.1  Standard  Openness  Questions 

2.1.1  Is  custodian  also  a  vendor  of  production  implementation? 

The  custodian  of  a  standard  could  be  a  public  standards  organisation  (such  as  IEEE,  OMG, 
etc.),  a  government  organisation,  private  consortium,  private  enterprise,  or  even  an 
individual.  It  is  conventional  to  assume  a  standard  maintained  by  a  public  standards  body 
is  "more  open"  than  one  that  is  maintained  by  a  proprietary  alternative.  However,  if  a 
standard  is  managed  appropriately,  there  is  no  justification  to  disregard  standards  that 
originate  from  a  proprietary  organisation  simply  because  of  their  origin. 

A  conflict  of  interest  can  arise  that  adversely  affects  openness  when  a  standard  managed 
by  an  organisation  that  is  also  a  commercial  vendor  of  an  implementation  of  that  standard. 
Safeguards  are  required  to  ensure  adequate  management  of  any  such  potential  conflict  of 
interest.  This  question  is  asking  whether  or  not  the  organisation  that  is  responsible  for 
managing  revisions  to,  and  distribution  of  the  standard,  is  also  offering  implementations 
of  the  standard  for  sale.  Note  this  does  not  include  the  provision  of  reference 
implementations  —  only  production  implementations  are  of  concern  at  this  point. 

2.1.2  Is  community-of-interest  established? 

If  the  organisation  managing  the  standard  is  also  producing  commercial  implementations 
of  the  standard,  it  is  imperative  that  processes  are  in  place  to  ensure  arbitrary  changes  are 
not  applied  to  the  standard  at  the  discretion  of  the  custodian.  It  is  important  that  an 
appropriately  constructed  community-of-interest  (COI)  is  responsible  for  planning  of 
revisions  to  the  standard.  If  a  standard  can  be  modified  without  regard  for  external 
vendors,  the  custodian  of  the  standard  has  an  unfair  advantage.  The  company  not  only  has 
the  opportunity  to  be  faster  to  market,  but  can  also  impose  a  level  of  control  on  its 
competitors.  Implementations  of  the  standard  are  therefore  likely  to  diverge  and 
competing  standards  (however  similar)  emerge.  It  is  therefore  critical  that  a  community- 
of-interest  governs  the  management  of  the  standard. 

2.1.3  Is  membership  open  and  equitable? 

Any  viable  CPO  must  not  place  unnecessary  restrictions  on  membership.  A  COI  must  give 
all  interested  parties  equal  right  to  shape  the  evolution  and  revision  of  the  standard.  There 
is  nothing  wrong  with  the  COI  charging  for  membership,  provided  the  membership  fees 
are  equivalent  for  equivalent  voting  rights.  Equally,  members  (or  indeed  the  custodian 
itself)  may  sponsor  the  COI  with  an  in-kind  contribution  in  lieu  of  a  financial  contribution. 

However,  every  effort  must  be  made  to  ensure  the  custodian  does  not  have  unfair 
influence  over  the  COI  decision  process. 

2.1.4  Is  governance  process  documented  and  followed? 

A  COI  process  needs  to  be  documented  in  order  to  instil  confidence  that  consistency  is 
being  achieved.  Similarly,  confirmation  that  the  process  is  being  followed  should  be 
sought.  This  could  be  achieved  by  contacting  a  random  sample  of  COI  participants  to 
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obtain  their  view  of  the  COI  process,  and  indeed  their  opinion  as  to  the  success  of  the  COI 
itself. 

2.1.5  Can  independent  vendors  implement  standard  and  deploy 
implementation? 

A  standard  must  be  able  to  be  independently  implemented  and  deployed  by  third-party 
vendors,  otherwise  the  standard  must  be  considered  proprietary.  If  blanket  or  selective 
legal  or  access  restrictions  are  imposed  that  prevent  vendors  deploying  their  own 
implementations  of  the  standard,  a  risk  exists  that  the  custodian  will  use  these  restrictions 
to  minimise  competition.  This  may  have  a  negative  impact  on  acceptance  and  life  span  of 
the  standard,  and  also  introduces  risks  associated  with  a  sole-source  relationship  (such  as 
long-term  cost  and  support). 

The  scope  of  access  is  important  however.  If  the  standard  is  intended  only  to  be 
implemented  and  used  within  a  well-defined  community  of  interest,  and  access  is  limited 
to  that  community  of  interest,  the  access  may  be  sufficient  within  the  intended  scope  to 
consider  the  standard  sufficiently  accessible.  An  example  could  be  a  standard  that  defines 
the  message  format  for  classified  data  being  passed  between  combat  platforms.  It  would 
be  unreasonable  to  force  the  standard  to  be  published  in  the  public  domain  if  it 
compromised  the  security  of  the  implementation.  However,  if  the  standard  was  freely 
available  to  vendors  that  had  a  need-to-know  (regardless  of  their  financial  or  political 
relationship  with  the  standard's  custodian),  then  access  would  be  considered  sufficient  in 
this  context. 

2.1.6  Is  standard  broadly  available  at  minimal  cost? 

Considering  the  intended  scope  for  access  of  the  standard,  is  the  standard  available 
without  an  unreasonable  financial  burden  to  any  vendor  that  wishes  to  access  the 
standard?  If  a  cost  is  charged  at  all,  the  cost  should  be  limited  to  the  reimbursement  of 
direct  expenses  incurred  in  delivering  the  standard  to  the  recipient  (such  as  shipping  and 
handling  charges).  Note  that  if  the  scope  of  intended  access  is  smaller  than  the  public 
domain  in  general,  the  availability  of  the  standard  should  not  be  restricted  within  the 
intended  scope.  Similarly,  if  a  cost  is  charged,  the  cost  should  be  consistent  regardless  of 
the  organisation  requesting  access. 

2.1.7  Are  production  implementations  available  from  different  vendors? 

A  standard  that  has  only  been  implemented  by  a  single  vendor  presents  a  number  of  risks 
to  an  end-user  including: 

•  certainty  of  supply 

•  confidence  that  the  implementation  will  continue  to  be  supplied  at  reasonable  cost. 

2.1.8  Is  a  reference  implementation  available  to  vendors? 

Broad  availability  of  a  reference  implementation  allows  vendors  to  verify  their  own 
implementations  for  compliance  against  a  common  reference  model.  It  also  simplifies  the 


4 


UNCLASSIFIED 


UNCLASSIFIED 

DST-Group-TN-1542 

process  of  implementing  the  standard,  so  potentially  encourages  competing 
implementations  to  be  developed,  and  for  those  implementations  to  be  compatible  with 
each  other. 

2.1.9  Are  implementations  interchangeable/  compatible? 

Compliance  to  the  standard  by  all  implementations  is  vital  to  the  integrity  of  the  standard 
itself.  However,  some  standards  (particularly  those  that  are  immature)  allow  vendors  too 
much  flexibility  which  causes  the  standard  to  either  be  extended  with  proprietary 
extensions,  or  implemented  in  incompatible  ways. 

There  are  three  possible  outcomes  from  this  question.  Either  the  majority  of 
implementations  are  compatible,  partially  compatible  or  predominantly  incompatible.  If 
implementations  are  partially  compatible,  it  is  important  to  determine  if  a  common 
compatible  core  "profile"  of  the  standard  is  consistent  amongst  implementations. 

2.1.10  Is  only  core  (compatible)  profile  required  for  target  deployment? 

Assuming  a  common  compatible  core  subset  (profile)  of  the  standard  could  be  determined 
to  be  compatible  amongst  different  implementations  of  the  standard,  determine  if  the 
profile  is  sufficient  for  the  needs  of  the  target  deployment.  If  so,  use  of  the  compatible 
profile  does  not  present  a  major  risk,  provided  the  incompatible  extensions  are  not 
permitted. 

2.1.11  Are  vendors  financially/ politically  independent? 

The  independence  of  different  vendors'  implementations  must  be  carefully  assessed  to 
ensure  a  monopolistic  or  oligopolistic  condition  does  not  exist.  An  attempt  should  be 
made  to  determine  if  any  financial  or  political  links  exist  between  any  pair  of 
implementation  vendors  to  determine  the  likelihood  of  a  conflict-of-interest  existing 
between  those  firms. 

If  the  opportunity  exists,  the  vendors  could  be  asked  to  declare  any  financial  or  political 
affiliation  they  may  have  with  other  implementation  vendors.  However,  this  is  unlikely  to 
be  practical  in  markets  where  the  customer  is  not  a  significant  participant. 

2.1.12  Has  the  quantity  of  vendors  significantly  decreased  in  the  past  3  years? 

A  decreasing  number  of  vendors  can  mean  either  the  market  is  maturing  and  less 
competitive  vendors  are  leaving  the  market  to  leave  only  the  strong,  viable  vendors.  Or,  it 
could  mean  that  the  standard  is  approaching  obsolescence  and  vendors  have  chosen  not  to 
continue  supporting  the  standard. 

In  order  to  determine  the  significance  of  a  decrease  in  the  number  of  vendors,  a  calculation 
using  the  Herfindahl-Hirschman  Index  [1]  (HHI)  can  be  used.  The  HHI  provides  an 
indication  of  market  concentration  and  is  calculated  by  summing  the  squares  of  the  market 
share  of  each  competing  vendor.  The  reciprocal  of  the  HHI  provides  an  indication  of  the 
"equivalent"  number  of  vendors  based  on  the  comparative  size  of  the  actual  vendors. 
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...where  n  is  the  number  of  vendors  in  the  market,  and  si  is  the  market  share  of  firm. 

For  example,  assume  12  companies  each  offered  an  implementation  to  an  emerging 
market,  and  thus  had  relatively  even  market  shares  of  between  6%  and  11%.  The 
equivalent  number  of  vendors  in  the  market  might  be  calculated  as: 


V 

equiv 

V 

equiv 


(0.1 12  +  2(0. 12)  +  2(0. 092)  +  3(0.082)  +  3(0.072)  +  0.062)  1 
11.66 


As  the  market  consolidated,  it  is  likely  that  a  market  leader  will  appear,  some  smaller 
vendors  will  cease  servicing  the  market,  and  the  remaining  vendors  will  absorb  the 
remaining  market  share.  For  example: 


V 

equiv 

V 

equiv 


(0.352  +  0.22  +  0.152  +  0.142  +  0.12  +  0.062)  ‘ 
4.58 


As  the  market  matures  further,  it  is  possible  that  one  or  two  vendors  will  essentially 
monopolise  the  market.  Even  though  a  number  of  smaller  vendors  are  available  within  the 
market,  the  equivalent  number  of  vendors  could  be  quite  small.  For  example: 


V 

equiv 


V 


equiv 


(0.52  +  0.45 2  +  0.022  +  0.01 2  +  0.01 2  +  0.01 2)  ' 
2.21 


When  the  equivalent  number  of  vendors  decreases  to  less  than  two,  questions  should  be 
asked  as  to  the  suitability  of  the  standard. 

2.1.13  Does  a  disruptive  technology  exist? 

Disruptive  technologies  can  rapidly  make  a  previously  popular  standard,  obsolete.  For 
example.  Voice  over  Internet  Protocol  (VoIP)  is  in  the  process  of  replacing  traditional 
dedicated  telephone  exchanges  and  wiring.  If  a  disruptive  technology  can  be  identified 
that  risks  replacing  the  standard  being  assessed,  a  migration  plan  is  essential. 

When  deciding  whether  or  not  a  disruptive  technology  is  relevant  to  the  standard  being 
assessed,  consider  emerging  variants  of  the  current  standard.  For  example.  Gigabit 
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ethernet  vs  10Mbps  ethernet.  Although  in  this  particular  case,  a  migration  path  was 
relatively  straight-forward,  there  may  be  cases  whereby  backward  compatibility  is  not 
maintained  and  therefore  presents  a  serious  risk  to  the  long-term  viability  of  the  current 
standard. 

2.1.14  Is  migration  to  alternative  standard  realistic? 

If  a  threat  to  the  current  standard  has  been  identified,  a  migration  path  is  needed  that  is 
realistic  in  terms  of  the  goals  of  the  target  deployment.  If  migration  from  the  current  to 
emerging  standard  is  likely  to  require  all  (or  a  substantial  number  of  different)  granule 
vendors  to  rework  their  implementation,  the  migration  is  unlikely  to  be  a  realistic  option. 

2.1.15  Have  changes  to  standard  occurred  in  last  2  years? 

If  the  standard  is  continually  evolving,  it  is  essential  that  its  evolution  does  not  force  users 
of  the  standard  to  repeatedly  re-engineer  their  solutions  in  order  to  keep  up  with  the  latest 
standard  version.  There  may  be  domains  in  which  a  2  year  assessment  is  inappropriate. 
However,  in  the  vast  majority  of  cases  for  computer-based  standards,  if  the  standard  has 
remained  unchanged  for  the  past  two  years,  it  can  be  considered  stable  and  most  vendors 
implementing  the  standard  are  likely  to  have  adopted  the  latest  release. 

2.1.16  Was  backward  compatibility  maintained? 

If  the  standard  has  changed  in  the  past  two  years,  it  is  necessary  to  determine  whether  or 
not  backward  compatibility  was  achieved.  A  standard  that  is  sufficiently  mature  to  allow 
incremental,  backwardly  compatible  change,  presents  substantially  less  risk  than  one 
where  users  must  re-engineer  their  solutions  in  order  to  keep  pace  with  new  revisions  of 
the  standard. 

2.1.17  Are  multiple  vendors  servicing  target  domain? 

If  the  primary  customer  domain  for  the  implementation  does  not  align  with  the  domain 
for  the  target  deployment,  evolution  of  the  standard  may  be  governed  by  the  needs  of  the 
primary  domain(s)  and  may  conflict  with  the  needs  of  the  target  application.  In  other 
words,  even  though  the  standard  may  appear  appropriate  for  use  in  your  target  domain  at 
the  present  time,  pressures  from  the  primary  domain  users  may  drive  changes  to  the 
standard  that  cause  it  to  diverge  from  your  own  goals. 

Additionally,  having  only  a  single  vendor  supporting  the  target  domain  implies  an 
effective  sole-source  relationship  between  the  vendor  and  customer,  and  thus  all  of  the 
risks  associated  with  such  a  relationship. 

2.1.18  Is  there  an  equity  of  scale  between  vendors? 

If  one  vendor  implementing  the  standard  is  substantially  larger  or  more  powerful  than  all 
other  independent  vendors,  there  are  several  minor  risks: 

•  smaller  vendors  could  be  forced  out  of  the  market  through  price  competition 
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•  smaller  vendors  could  be  forced  out  of  the  market  due  to  the  larger  vendor  driving 
evolution  of  the  standard  more  quickly  than  can  be  supported  by  the  smaller 
vendors 

•  the  larger  vendor  could  use  their  market  position  to  introduce  extensions  to  the 
standard  that  do  not  align  with  the  goals  of  the  community  of  interest  —  the 
standard  may  therefore  diverge  into  official  and  de-facto  versions 

•  smaller  vendor(s)  could  be  easily  acquired  by  the  larger  vendor,  thereby  forming  a 
monopoly.  The  HHI  (see  Section  2.1.12)  could  be  used  here,  although  the  question 
is  of  equity  rather  than  quantity,  so  it  may  be  appropriate  to  divide  the  number  of 
vendors  by  the  HHI  and  determine  equality  based  on  how  close  the  result  is  to  1.0. 

2.1.19  Is  standard  used  in  multiple  distinct  domains? 

A  standard  that  has  gained  acceptance  in  multiple  domains  is  more  likely  to  resist  domain 
specific  pressures  that  adversely  affect  the  suitability  of  the  standard  to  the  target  domain 
that  could  otherwise  be  introduced  as  the  standard  evolves. 
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